ChordMR: A P2P-based Job Management Scheme in Cloud

MapReduce is a programming model and an associated implementation for processing parallel data, which is widely used in Cloud computing environments. However, the traditional MapReduce system is based on a centralized master-slave structure. While, along with the increase of the number of MapReduce jobs submitted and system scale, the master node will become the bottleneck of the system. To improve this problem, we have proposed a new MapReduce system named ChordMR, which is designed to use a peer-to-peer Chord network to manage master node churn and failure in a decentralized way. More importantly, we propose a job management scheme which contains the aspects of job assignment, job backup, job recovery, etc. In addition, in ChordMR, the job backup scheme makes a number of 2 -1 1 K identical copies for each job and classifies the jobs into K2 classes to achieve the goals of fault tolerance and load balance. The theoretical and simulation results show that our ChordMR is effective.